Software testing is a primary process to determine errors and failures in order to improve software quality and reliability. However, software testing is the most expensive process in terms of finance, resources and time. Generally, testing consumes major part of the total cost of a software development project. The companies with high process maturity levels and excellent measurement data are also unable to address the issue of high cost of testing. Schedule over-runs and underestimation of testing resources are also observed in many surveys across organizations. One of the reasons for over-run of their schedule is lack of testing tools. Test automation is one of the main approaches to reduce time and potential human error in testing process and decrease overall testing costs.
Several studies show test automation is gradually gaining due importance in the software industry. Software testing involves identifying suitable testing methods, techniques or criteria and generation of test cases using identified criteria. Automation can be possible on all testing tasks but it is usually cost-effective for repetitive tasks, such as unit testing and regression testing, in which test cases are executed every time changes are made. Cost-effectiveness of automation increases every time an automated test case is executed.
Most of the potential benefits of automated test execution will be lost if the success or failure of test cases is not assessed. A test output evaluation may help in evaluating the automated test execution to determining passing or failure of the test case by comparing the actual and expected output. A test output evaluation is an essential part of test execution because a wide range and large number of test cases are executed many times and the behavior must be checked for every test case every time.
Though, the current software testing environments provide automated test execution. However, the test output evaluation is being performed manually. Therefore, there is a general need for a method and a system for automating test output evaluation. Several aspects of the present disclosure discloses a method and a system for automating test output evaluation as described in details in below sections.